Monday morning, the dashboard is green. The new checkout button, variant B, "converted 12% better" over the weekend. Someone wants to roll it out to everyone and put it in the quarterly slides. Before anyone does, a marketing analyst asks four quiet questions. How many people actually saw each button? Could a lift this size appear when the two buttons are identical? Who was in the test, and who was missing? And is "conversion" still the thing the company wants, or just the thing it counts?
Those four questions are most of this subject. Marketing research is the craft of finding out what customers think, want and do, without fooling yourself on the way. It sits between statistics and psychology, and it is the course where the statistics you learned meet people who do not behave like coin flips: they misread your question, skip your survey, and change what they do once they are measured.
A survey question is not a window onto what people think. It is a nudge, and people answer the question they hear. A famous demonstration: Elizabeth Loftus and John Palmer showed people films of car crashes in 1974 and asked how fast the cars were going when they "smashed into" each other. Other groups heard "hit" or "contacted". The more violent the verb, the faster people said the cars had been going. The wording did not change the film. It changed the memory report.
Researchers keep a short list of ways a question goes wrong. A leading question smuggles in the answer ("How satisfied are you with our excellent service?"). A double-barrelled question asks two things at once, so a "no" cannot be read ("Is the price and quality good?"). A loaded option set has more positive choices than negative ones. And a hypothetical question ("Would you buy this?") invites a cheerful yes that real wallets rarely honour. The fix is boring and reliable: ask about what people did (last purchase, last week), one thing at a time, with balanced options, in plain words, and test the draft on five people before sending it to five thousand.
Four pairs. Tap the wording a careful researcher would send.
Surveys can only measure what you already thought to ask. Before you count anything, you listen: one-to-one interviews, watching someone use the product, a focus group, a diary study. This is qualitative research. It gives you words, not percentages, and its job is to find the questions worth putting in the survey and the hypotheses worth testing. A bakery that "knows" customers want cheaper bread might hear in three interviews that the real problem is that the bread goes stale by Tuesday.
How many people is enough? Usability researchers model it simply: if each tester independently runs into any given problem with probability p, then n testers find a share 1 − (1 − p)n of the problems. Jakob Nielsen and Thomas Landauer's 1993 paper put the average p near 0.31 for usability problems, which is where the famous "five users find about 85% of the problems" rule comes from. Treat that as a rule of thumb for obvious problems in one product, not a law: rare problems and different user types need more people, and a focus group's loudest voice can bury the rest.
A model, not a measurement. 0.31 is the average Nielsen and Landauer reported; your product and your users will differ.
You cannot ask everybody, so you ask a few and infer the rest. A properly random sample of n people gives an estimate of a proportion that is off by roughly this much, 19 times in 20:
For n = 1,000 and p = 0.5 that is ±3.1 points. Quadruple the sample to 4,000 and the margin only halves, to ±1.5, because n sits under a square root. That is why national polls stop at around a thousand: after that, money buys little.
The formula hides a trap. It assumes the sample is random: that everyone had the same chance of ending up in it and answering. When that fails, a bigger n only makes you more confident in the wrong number. In 1936 the magazine Literary Digest mailed about 10 million ballots and got about 2.38 million back, a vast sample, and predicted that Alf Landon would beat Franklin Roosevelt with about 57% of the vote. Roosevelt won with 60.8%. The mailing lists came from its own readers, car owners and telephone subscribers, who in the Depression were richer than average, and later research pointed to non-response as the main cause of the size of the miss: people angry at Roosevelt were more likely to send their ballot back.
Each bar is a poll's 95% interval. The dashed line is the truth. Illustrative simulation, not real data.
With no bias and 257 respondents, about 38 of the 40 bars cover the truth. Raise the sample to 3,000 with a 1.4 response bias and the bars get tight and none of them covers the truth. Precision and accuracy are different things: sample size cures noise, never bias.
An A/B test shows version A to a random half of visitors and version B to the other half, then asks whether the difference in conversion is more than luck. Randomising is what makes it a real experiment: any other difference between the groups is chance, so a difference in outcome can be pinned on the change. The test statistic is a z-score: the observed gap divided by the gap's typical size from noise alone. Past ±1.96, a gap this big would occur by luck about 1 time in 20 if the versions were identical (p < 0.05).
Notice what "1 in 20" means. Run 20 tests on changes that do nothing and you expect one to "win". Now add the common mistake, peeking: you look at the dashboard every day and stop the moment it turns significant. Each look is another chance for luck to cross the line, and the false-alarm rate climbs well above 5%. Statisticians have known this since sequential analysis in the 1940s; online experimentation books still list it as the number-one error. Run an A/A test (two identical versions) below and watch.
Each experiment simulates real visitors with a seeded random generator, then applies a two-sided z-test at 5%. Illustrative simulation.
With identical versions and one look, about 5% are flagged: that is the promise. With ten looks, roughly 18% are. And with a real lift of 10% on a 5% baseline and 2,000 visitors per version, most tests miss the real effect: small samples have low power. Good practice is to decide the sample size in advance from the smallest lift worth caring about, run to that size, and read once; or use a method designed for continuous monitoring.
Ask "which of these features matters to you?" and everyone says all of them, especially the cheap price. Real choices are trade-offs, so conjoint analysis forces them. You describe products as bundles of attributes (price, battery life, waterproofing), show people many bundles, and have them rate or choose among them. Because the bundles vary in a planned way, a little arithmetic pulls out how much each level contributed to liking: its part-worth. The method comes from mathematical psychology (Luce and Tukey, 1964) and reached marketing in the 1970s. Today's version mostly shows pairs or triples and asks for a pick, but the logic is the same.
Below, a made-up person has rated eight portable speakers from 0 to 10. Two prices, two battery lives, waterproof or not: all eight combinations, which makes the design balanced, so each part-worth is simply the average rating when an attribute is at one level minus the average at the other. The ratio of a feature's effect to the price effect turns it into dollars.
Drag the sliders to your own taste. Importance = an attribute's range of effect ÷ the sum of ranges. Dollar value = feature effect ÷ (price effect per dollar). Illustrative, eight profiles only.
Try the "feature-lover": the dollar values come out far above the price range tested. That is the classic warning: conjoint works inside the range you tested and stretches badly outside it. It also measures stated choices; it does not know what happens on a crowded shelf, so serious studies check it against real sales.
"The customer" is a fiction. A gym has early-morning regulars and January joiners; a supermarket has stock-up families and lunchtime grazers. Segmentation splits a market into groups who respond alike, so you can serve each differently. When you have data on behaviour, a standard method is k-means clustering: pick k, drop k centres at random, assign every customer to the nearest centre, move each centre to the average of its customers, and repeat until nothing moves.
Two facts to remember. First, you choose k, and the data will always give you k groups, real or not. The honest check is the "elbow": how much tighter the groups get as k rises. Second, the result depends on where the centres start; a bad start can settle on a worse split. The algorithm finds clusters; a human has to decide whether they mean anything, and give them names like "bargain hunters" that a campaign can act on.
90 invented customers. Horizontal: visits per month. Vertical: spend per visit. Spread (inertia) = total squared distance to each group's centre.
A dashboard turns a messy business into a few numbers, and then people steer by them. In 1975 the British economist Charles Goodhart noted that a statistical regularity tends to collapse once pressure is put on it for control purposes. The anthropologist Marilyn Strathern put it in the form everyone quotes in 1997: when a measure becomes a target, it ceases to be a good measure. Goodhart himself later said he meant it as a throwaway remark.
The marketing version is everywhere. Reward a team for clicks and headlines get more tempting and less honest. Reward a bank's staff for accounts opened and, in the best-documented case, they open accounts nobody asked for: in September 2016 US regulators announced $185 million in penalties against Wells Fargo after its own analysis found employees may have opened more than two million deposit and credit card accounts without authorisation, to meet sales targets and earn bonuses. The measure, accounts per customer, had become the target.
The defence is a habit, not a formula: pair every metric with one it can't easily fake (clicks with purchases, purchases with returns and complaints), look at the distribution as well as the average, and ask what you would do to hit this number if you did not care about the customer.
A stylised model: curiosity headlines raise clicks (1,000 × (1 + 1.5x)) but the share of clickers who buy falls (6% × (1 − 0.85x²)), with x the pressure from 0 to 1. The shape is the lesson, not the numbers.
Evidence is made, not found. The wording of the question, the way the sample was drawn, the number of times you looked, and the target you set all shape the number before you see it. A marketing analyst's real skill is asking, calmly and before the meeting, how each of those shaped this one.
Try this: the next time you see a poll, a review score or an "up 12%" claim, find three things: how many people, how they were chosen, and what exactly was asked or counted. If you can't find them, treat the number as decoration.
Sources. Margin of error, z-tests, the A/B simulation and k-means are standard statistics, computed here and checked in code. Loftus and Palmer, "Reconstruction of automobile destruction", Journal of Verbal Learning and Verbal Behavior (1974). Nielsen and Landauer, "A mathematical model of the finding of usability problems", INTERCHI 1993. The 1936 Literary Digest figures are from the Wikipedia article on the magazine, which summarises later analyses. Luce and Tukey, "Simultaneous conjoint measurement", Journal of Mathematical Psychology (1964). Goodhart (1975) and Strathern, "'Improving ratings': audit in the British University system", European Review 5(3), 1997. The Wells Fargo figures are from the US Consumer Financial Protection Bureau's press release of 8 September 2016. All datasets in the interactives are invented for illustration.